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Abstract  Many  problems  of  mathematical  physics  can  be  formulated  in  terms 
of  the  operator  equation  Ax  =  y,  where  A  is  an  integro-dif ferential 
operator.  Given  A  and  x,  the  solution  for  y  is  usually  straightforward. 
However,  the  inverse  problem  which  consists  of  the  solution  for  x  when 
given  A  and  y,  is  much  more  difficult.  In  this  paper  the  following 
questions  relative  to  the  inverse  problem  are  explored: 

(1)  Does  specification  of  the  operator  A  determine  the  set  (y) 
for  which  a  solution  x  is  possible? 

(2)  Does  the  inverse  problem  always  have  a  unique 
solution? 

(3)  Do  small  Derturbations  of  the  forcing 

function  y  always  result  in  small  perturbations  of  the  solution? 

(4)  What  are  some  of  the  considerations  that  enter  into  the  choice 
of  a  solution  technique  for  a  specific  problem? 

The  concept  of  an  ill-posed  problem  versus  that  of  a  well-posed 
problem  is  discussed.  Specifically,  the  manner  by  which  an  ill-posed 
problem  may  be  regularized  to  a  well-posed  problem  is  presented.  The 
concepts  are  illustrated  by  several  examples. 
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1.  Introduction;  Many  problems  of  mathematical  physics  can  be  formulated 
in  terms  of  an  operator  equation 

Ax  =  y.  (1) 

Typically,  y  denotes  the  system  output  and  x  denotes  the  unknown  to  be 
solved  for.  Depending  upon  the  problem,  either  A  or  x  may  be  defined 
as  the  system  operator.  For  example,  in  a  linear  system  identification 
problem  the  goal  is  to  determine  the  system  operator  given  the  response 
to  a  specific  input.  The  system  operator  (i.e.  the  impulse  response) 
would  then  be  denoted  by  x  and  the  input  by  A.  On  the  other  hand,  in  a 
radiation  problem  the  objective  might  be  to  solve  for  the  excitation 
currents  on  a  structure  given  the  radiated  field  y(t).  In  this  case,  the 
excitation  current  would  be  denoted  by  x  and  the  system  operator  (i.e. 
the  Green's  function  operator)  would  be  denoted  by  A.  Given  A  and  x,  the 
direct  problem  of  solving  for  the  output  y  is  relatively  straightforward. 
However,  the  inverse  problem  which  consists  of  the  solution  for  x  given 
A  and  y  is  much  more  difficult.  One  of  the  difficulties  that  may  arise 
is  illustrated  in  Example  1. 

Example  1:  Consider  a  linear  system  with  impulse  response  x(t)  that  is 
excited  by  an  input  A(t)  applied  at  t  =  0.  The  system  response  is  given 
by  the  integral  equation 

t 

y(t)  =  Ax  =*  j  A(t  -  t)x(t )dx *  (2) 

0 

Given  the  Input  A(t)  and  the  output  v(t),  assume  the  objective  is  to  find 
the  Impulse  response  x(t).  In  spite  of  the  fact  that  a  solution  for  x(t) 
may  exist  and  be  unique,  the  integral  equation  of  (2)  is  difficult  to 
solve  due  to  the  smoothing  action  of  the  convolution  operator.  For 
example,  denote  the  exact  solution  by 

x^(t)  -  x (t )  (3) 

and  a  perturbed  solution  by 

X£(t)  ■  x(t)  +  C  sin  ait  (4) 
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Even  for  very  large  values  of  C,  the  frequency  oj  can  be  chosen  high 
enough  such  that  the  difference  between  y^  =  Ax^  and  y^  =  AX2  is  made 
to  be  arbitrarily  small.  This  is  demonstrated  as  follows: 

The  difference  between  y^  and  y^  is 

t 

y,  -  y,  =  Ax-  -  Ax  *  C  [A(t  -  T)sin  wt  df,  <5) 

4  i  t  1  j 

0 


If  the  input  A  is  bounded. 


it  follows  that 


I  (A  |  |  (a  constant) 


y2  -  yx  1  cm 


sin  toT  dT 


CM(1  -  cos  wt) 


U) 


We  conclude  that 


(6) 

(7) 


(8) 


Obviously,  by  selecting  oj  to  be  sufficiently  large,  the  difference  y ^  ~ 
can  be  made  arbitrarily  small.  The  ill-posedness  of  this  example  is 
evidenced  by  the  fact  that  small  differences  in  y  can  map  into  large 
differences  in  x.  This  is  a  serious  problem  because,  in  practice,  measure¬ 
ment  of  y  will  be  accompanied  by  a  nonzero  measurement  error  (or  repre¬ 
sentation  error  in  a  finite  dimensional  digital  system)  <5.  Use  of  the 
"noisy"  data  can  yield  a  solution  significantly  different  from  the  desired 
solution. 

When  a  problem  is  ill-posed,  an  attempt  should  be  made  to  regularize 
the  problem.  The  solution  to  the  regularized  problem  will  be  well- 
behaved  and  will  offer  a  reasonable  approximation  to  the  solution  of  the 
ill-posed  problem. 

The  concepts  of  well-posed  problems  and  the  regularization  of  ill- 
posed  problems  have  been  discussed  in  depth  in  the  mathematical  litera¬ 
ture  [1  -  8J.  In  this  paper,  we  illustrate  some  of  the  significant 
concepts  involved  by  means  of  simple  examples  that  arise  in  engineering 
applications. 
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2 .  The  Definition  of  a  Well-Posed  Problem 

Hadamard  introduced  the  notion  of  a  well  posed  (correctly  or  properly 
posed)  problem  in  the  early  1900's  when  he  studied  the  Cauchy  problem  in 
connection  with  the  solution  of  Laplace's  equation.  He  observed  that  the 
solution  x(t)  did  not  continuously  depend  on  y(t).  Hadamard  concluded 
that  something  had  to  be  wrong  with  the  problem  formulation  because 
physical  solutions  did  not  exhibit  this  type  of  discontinuous  behavior. 
Other  mathematicians,  such  as  Petrovsky,  also  reached  the  same  conclusion. 
As  a  result  of  their  investigation,  a  problem  characterized  by  the  equation 
Ax  =  y  is  defined  to  be  well-posed  provided  the  following  conditions  are 
satisfied 

(1)  The  solution  x  exists  for  each  element  y  in  the  Hilbert  space  Y 

(2)  The  solution  x  is  unique 

(3)  Small  perturbations  in  y  result  in  small  perturbations  in  the 
solution  x  without  the  need  to  impose  additional  constraints. 

If  any  of  the  above  conditions  are  violated,  the  problem  is  said 
to  be  ill-posed.  Examples  of  ill-posed  problems  are  now  discussed.  We 
first  consider  an  example  in  which  condition  (1)  is  violated. 


Example  2.  Suppose  it  is  known  that 

y  ■  Xjtj  +  x2t2  .  (9) 

In  addition,  assume  that  the  observation  y  is  contaminated  bv  additive 
noise.  The  observations  and  corresponding  time  instants  are  tabulated 
in  Table  1. 

Table  1 

tx  2  11 

t2  12  3 

observation  of  y  3  3  5 


Determination  of 
matrix  equation: 

Ax  - 


and  x2  leads  to 


consideration  of  the  following 


3 

3 

5 

L.  J 


(10) 


M. 
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Since  (10)  consists  of  3  Independent  equations  in  2  unknowns,  a  solution 
does  not  exist.  Condition  (1)  is  violated  and  the  problem  is  ill-posed. 


Example  3:  Violation  of  condition  (2)  is  illustrated  in  this  example. 

Given  a  set  of  measured  sample  values,  consider  the  problem  of  using 
Prony's  method  to  represent  a  time  function  y(t)  by  a  sum  of  two  complex 
exponentials.  Specifically,  it  is  desired  to  determine  8^  and  C^,  i  = 

1,2,  such  that 

expC^t)  +  C2  exp(82t)  =  y(t)  (11) 


In  this  case  the  unknown  vector  is 


(12) 


and  A  is  the  operator  that  maps  x  into  v.  Assume 

y (t)  *  exp  [-0.00035t]  cos  (0,25t). 
Corresponding  to  sampling  instants  given  by 

t  -  (0,1, 2, 3, 4) 

y(t)  is  given  by 


(13) 


(14) 


y  -  (1,0.97,0.88,0.73,0.54)  ,  (15) 

However,  in  practice,  measurements  of  the  sample  values  are  accompanied  by 
measurement  error.  For  example,  the  measured  sample  values  of  y  may  be 
given  by 

d  -  (dQ,d1,d2,d3,d4)  -  (1.01,0.96,0.89,0.72,0.54) 


(16) 


Employing  Prony's  method  [9],  it  is  convenient  to  define 

si  -  expfg^  ;  i  *  1,2.  (17) 

Values  of  s^  are  then  obtained  from  the  polynomial  equation 

3qS^  +  a^s^  +  a2  »  0.  (18) 

Using  a  least-squares  error  approach  [10],  the  coefficients  in  (18)  are 
obtained  by  solving  the  matrix  equation 
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C  a  - 


where 


c00  C10  °20 

ao 

r  n 

0 

c01  C11  c21 

31 

2 

0 

C02  C12  C22 

a2 

0 

■a  . 

4 

C  »  F  d.  d  =  C  . 

ij  k£2  Vi\-j  Cji* 


(19) 


(20) 


In  general,  the  determinant  of  C  is  nonzero  and  a  solution  does  not  exist 
to  (19).  However,  note  that  any  one  of  the  coefficients  in  (18)  can  be 
set  equal  to  unity  without  affecting  the  solution  for  the  roots. 

Depending  upon  whether  ag,  a^  or  a2  Is  chosen  to  be  unity,  (19)  reduces  to 
one  of  the  following  sets  of  equations: 


(21a) 


(21b) 


(21c) 


Therefore,  having  set  one  of  the  coefficients  to  unity,  the  remaining 
two  coefficients  may  be  determined  from  either  (21)  a,  b  or  c.  These 
coefficients  are  then  substituted  in  (18)  which  is  then  solved  for  s^. 
Unfortunately,  (21a),  b  or  c  do  not  result  in  an  equivalent  set  of 
coefficients.  Consequently,  the  values  of  s^  differ  as  shown  in  Table  2. 


Table  2 


Coefficient  set  to  unitv 

Equation  used 

li 

ao“  1 

(21a) 

0.87  +  jO.23 

al  "  1 

(21b) 

0.98  +  j0.22 

*2  *  1 

(21c) 

1.1  +  jO.14 

Since  the  solution  is  not  unique,  condition  (2)  is  violated  and  the  prohlor 
is  ill-posed  when  inaccurate  measured  data  is  used.  Interestingly  enough. 


had  Che  samples  of  y  In  (15)  been  available  In  place  of  the  measured 
daca  d  in  (16),  an  equivalent  set  of  coefficients  would  have  been  obtained 
from  either  (21a),  b  or  c.  Use  of  these  coefficients  in  (18)  results  in 
s^  *  0.97  +  j0.25,  where  the  numerical  values  have  been  rounded  off  to 
two  decimal  places.  Therefore,  Prony's  method  is  well-posed  in  the  absence 
of  any  disturbance. 

The  final  example  in  this  section  illustrantes  a  problem  for  which 
condition  (3)  is  violated. 


Example  4:  Consider  solution  of  the  matrix  equation  Ax  *  y  where  A  is 
the  4x4  symmetric  matrix  of  reference  [11]. 


and 


36.86243 

51.23934 

53.50338 

50.49425 


51.23934 

71.22350 

74.37005 

70.18714 


53.50338 

74.37005 

77.66275 

73.29752 


50.49425 

70.18714 

73.29752 

69.17882 


T 

y  * 


[192.09940  267.02003  278.83370  263.15773 


(22) 


(23) 


where  the  superscript  T  denotes  transpose.  It  is  readily  verified  that 
the  exact  solution  is  given  by 

xT  =  [1  1  1  I], 

However,  it  can  be  shown  that  A  is  an  ill-conditioned  matrix  in  the  sense 
that  the  ratio  of  its  maximum  to  minimum  eigenvalues  is  on  the  order  of 

io18. 


Let  us  study  the  effect  on  the  solution  x  when  only  one  component  of  y 
is  changed  in  the  fifth  decimal  place.  Specifically,  we  obtain  the  results 
presented  in  Table  3. 

Table  3 


J. 

y  -*■ 

[192.00939 

267.02003 

278.83370 

263.15773] 

T 

x  -*■ 

[-6,401,472,429 

3,866,312,299 

1,607,634,613 

-953,521,374] 

T 

Y  -<■ 

[192.09940 

267.02002 

278.83370 

263.15573] 

T 

x  ■* 

[3,866,312,299 

-2,335,145,694 

-970,966,842 

575,900,539] 

T 

y  -*■ 

[192.00940 

267.02003 

278.83369 

263.15573 

T 

x  -*• 

[1,607,634,615 

-970,966,842 

-403,733,529 

239,462,717] 

•< 

1  H 

♦ 

[192.09940 

267.02003 

278.83370 

263.15772] 

T 

x 

[-953,521,374 

574,900,539 

239,462,717 

-142,030,294] 

Clearly,  extremely  small  perturbations  in  y  result  in  large  variations 
in  x.  Condition  (3)  is  violated  and  the  problem  is  ill-posed. 
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3.  Regularization  of  an  ill-posed  problem 

Most  inverse  problems  of  mathematical  physics  are  ill-posed  under 
the  three  conditions  of  Hadamard.  In  a  humorous  vein  Stakgold  [12] 
pointed  out  that  there  would  likely  be  a  sharp  drop  in  the  employment  of 
mathematicians  if  this  were  not  the  case. 

Given  an  ill-posed  problem,  various  schemes  are  available  for  defining 
an  associated  problem  which  is  well-posed.  This  approach  is  referred  to  as 
regularization  of  the  ill-posed  problem.  In  particular,  an  ill-posed 
problem  may  be  regularized  by 

(a)  changing  the  definition  of  what  is  meant  by  an  acceptable  solution, 

(b)  changing  the  space  to  which  the  acceptable  solution  belongs, 

(c)  revising  the  problem  statement, 

(d)  introducing  regularizing  operators,  and 

(e)  introducing  probabilistic  concepts  so  as  to  obtain  a  stochastic 
extension  of  the  original  deterministic  problem. 

The  above  techniques  are  now  illustrated  by  a  series  of  examples. 

Technique  (a)  i s  demonstrated  in  Example  (5). 

Example  5.  Once  again  consider  the  problem  introduced  in  Example  2.  This 
resulted  in  (10)  for  which  a  solution  did  not  exist.  Nevertheless,  an 
approximate  solution  is  possible.  One  of  several  possible  approximate 
solutions  for  any  over determined  linear  system  is  the  Moore  Penrose 
generalized  inverse  [25]  which  is  a  least  squares  solution  to  (1)  and  is  given  by 

x  „  [a*A]_1  A*y  =  [J;31422.  (25) 

v  '  •  the  asterisk  denotes  the  transpose  conjugate.  Note  that  x  does  not 
exactly  satisfy  any  of  the  three  equations  in  (10).  Yet  x  is  a  reasonable 
approximate  solution.  We  see  that  the  original  problem  which  was  ill-posed 
because  a  solution  did  not  exist  has  been  regularized  by  redefining  what  is 
meant  by  an  acceptable  solution. 

Use  of  technique  (b)  is  illustrated  in  Example  (6). 

Example  6:  Relative  to  Lewis  Bojarski  inverse  scattering  [13-15],  the 
relationship  at  all  frequencies  and  aspects  between  the  back  scattered 
fields  y(k)  to  the  size  and  shape  of  the  target  x(r)  is  given  by 
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x(r)exp(-jkr)dr. 

00 


(26) 


This  relation  is  valid  under  the  assumption  that  the  target  currents  are 
obtained  using  a  physical  optics  approximation.  In  (26)  A  is  the  Fourier 
transform  operator.  The  solution  to  (26)  is  given  by  the  inverse  Fourier 
transform  00 

x(r)  *  -2-  [  y (k)exp[ jkrjdk  (27) 

fa  ' 

—00 

where  for  convenience,  the  constant  Q  *  1//?tt  has  been  introduced. 

It  is  now  demonstrated  that  this  problem  is  ill-posed  when  solutions 
are  allowed  in  the  metric  space  for  which  the  norm  is  defined  to  be 

||x1(r)  -  xQ(r)|j  »  maxjx1(r)  -  xQ(r)|.  (28) 


On  the  other  hand,  the  problem  is  well  posed  in  the  space  for  which  the 
norm  is  defined  to  be 

1/2 

.  /  V  /  V  II  U  r  I  r  /  *  /  VI4*.  ' 

|X 


a  r  2 

^(r)  -  xQ(r)Sl  =  {  I  [x^(r)  -  XQ(r)]  dr} 


(29) 


—CO 


The  first  norm  is  referred  to  as  the  'sup'  norm  whereas  the  second  norm 
is  known  as  the  «£,2  norm. 

Given  a  scattered  field  function  yQ(k),  consider  the  perturbed 
scattered  field  function  y^(k)  such  that 

2 

y^k)  »  y0(k)  +  e  exp[-ak  1 
where  e  and  a  are  arbitrary  positive  constants.  Then 

x,  (r)  *  -^-Texpt  jkr](yn(k)  +  e  exp[-ak]2)dk 
1  fin  >  u 


n(r)  +  ^  |°°exp[+jkr  -ak2]dk 
0  fin  i 


-  *o(r)  f"fc  '• 


(30) 


Using  the  sup  norm,  the  perturbations  in  the  expressions  for  y^(k)  and 
x^(r)  are  bounded  by 
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max  ||y,  (k)  -  yQ(k)  ||  <  e 
k  1  u 

max  ||x  (r)  -  xQ(r)||  <.-2^-  (31) 

r  /2a 

Obviously,  for  a  fixed  small  perturbation  in  y^(k),  the  perturbation  in 
Xg(r)  can  be  arbitrarily  large  depending  upon  the  choice  of  a.  Hence, 
the  third  condition  of  Hadamard  is  violated  and  the  inverse  Fourier 
transform  is  ill-posed  in  the  metric  space  using  the  sup  norm.  Physi¬ 
cally,  this  implies  that  the  inverse  Fourier  transform  is  ill-posed 


under  a  pointwise  convergence  criterion. 

2 

On  the  other  hand  using  the  norm, 

00 

||yi<k)-y0<k)||  -  {|(y1(k)-yo(k)]2dk} 


1/2 


e2exp[-2ak2]dk}1/2 


and 


f°°  2  2  f  2 

l|x^(r)  —  xQ(r)  ||  =  {jTx1(r)-x0(r)]2dr}1/,2  *  (^^-Jexp dr) 

— oo 


1/2 


_  ,  it  .1/4 


(32) 


2 

Since  the  c£_  norm  of  the  perturbation  for  x^Cr)  and  y^Ck)  differ  only 
by  the  constant  Q,  small  perturbations  in  yg(k)  result  in  small  per¬ 
turbations  in  Xg(r).  Hence,  the  inverse  Fourier  transform  is  well  posed 
2 

under  the  norm,  which  implies  that  convergence  in  the  mean  square 


for  the  shape  of  the  structure  can  be  obtained. 

In  this  example,  we  have  illustrated  how  a  problem  can  be  made 

well-posed  just  by  changing  the  soace  of  the  solution.  Phvsically 
this  implies  that  the  problem  is  well  posed  when  we  want  an  "average" 
description  of  the  target  and  the  problem  is  ill  posed  when  we  demand  an 
accurate  precise  definition  of  the  target. 
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Example  7;  In  this  example  a  problem  is  regularized  by  means  of 
technique  (c)  (i.e.,  revising  the  problem  statement).  Consider  a 
linear  system  identification  problem  where  A(t),  x(t)  and  y(t)  denote 
the  input,  impulse  response,  and  output,  respectively.  Using  the 
convolution  integral  the  output  is 

y(t)  «  j°°x(t  -  T)A(T)dT. 

— oo 

Application  of  Fourier  transform  theory  results  in  the  solution  for 
the  impulse  response  as 


x(t)  *  J--  [  iiiiil  exp  (j(jJt)dtD 
27T  J  if, .A 


where  y(w)  and  A(uj)  are  the  Fourier  transforms  of  y(t)  and  A(t), 
respectively. 

In  practice,  the  ideal  output  is  contaminated  by  additive  noise. 
Denote  the  measured  response  by  d(t)  where 

d(t)  -  y (t)  +  n(t)  (j 

and  n(t)  represents  a  stationary  zero  mean  additive  noise  process.  The 
correlation  function  of  n(t)  is  denoted  by  <j> (x )  and  its  Fourier  trans¬ 
form  by  $((*)).  When  the  measured  noisy  response  d(t)  is  used  in  place 
of  the  ideal  output  y(t),  the  solution  x(t)  is  a  random  process.  The 
variance  of  x(t)  is  given  by  [16] 


j~  r_i 
2*  J  u 


|a(u)|‘ 


This  problem  is  considered  to  be  well  posed  only  if  a*  is  suitably  small. 

Typically,  the  noise  is  assumed  to  contain  a  background  component  of 

white  noise.  With  this  assumption,  the  power  spectral  density  <Kw) 

approaches  a  nonzero  constant  K  as  |u|  approaches  infinity.  On  the 

other  hand,  for  finite  energy  inputs,  |A((*))|  approaches  zero  as  ] u> [ 

2 

approaches  infinity.  With  these  assumptions,  o x  Is  infinite  and  the 
problem  is  obviously  ill  posed. 


'*;■  ■  \.n. :s tv  - C 
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This  problem  can  be  made  well-posed  by  revising  the  problem  state¬ 
ment.  From  a  mathematical  point  of  view,  the  input  could  be  restricted 
to  signals  whose  Fourier  transform  A(oj)  is  a  rational  function  with 

numerator  polynomial  of  higher  degree  than  the  denominator  polynomial. 

2 

Then,  even  with  a  white  noise  background,  remains  finite  and  the 
problem  becomes  well-posed.  However,  this  regularization  implies  the 
presence  in  A(t)  of  singularity  functions  such  as  impulses,  doublets,  etc. 
Therefore,  the  regularization  is  achieved  at  the  expense  of  requiring 
A(t)  to  be  an  unrealizable  signal  having  infinite  energy. 

A  preferable  approach  is  to  remove  the  commonly  used  assumption 
of  a  white  noise  background.  Provided  the  power  spectral  density  of 
the  noise  <t>(u)  falls  o|f  in  frequency  at  a  rate  which  is  at  least  as 
fast  as  that  of  (£>0»  arbitrary),  <J~  will  be  finite.  This  demon¬ 

strates  that  the  noise  must  be  carefully  modeled  if  the  problem  is  to  be  well-posec 
Technique  (d),  the  application  of  regularizing  operators,  is  dis¬ 
cussed  next.  In  example  2,  it  was  shown  that  the  three  simultaneious 
equations  were  inconsistent  and,  therefore,  an  exact  solution  did  not 
exist.  In  example  5,  an  approximate  solution  was  obtained  using  the 
least  squares  approach.  The  solution  was  given  by 

x  ■  [A*A]”^  A*y  .  (37) 

As  the  dimensionality  of  the  problem  is  increased,  there  is  a  tendency 
for  the  solution  to  oscillate  and  increase  in  magnitude.  Therefore,  the 
problem  becomes  ill-posed  as  the  dimensionality  increases. 

This  difficulty  may  be  overcome  by  introducing  regularizing  operators 
which  impose  additional  constraints  on  the  solution.  In  particular, 
instead  of  solving  the  problem 

Ax  -  y  (38) 

2 

attention  is  focused  on  the  problem  of  minimizing  | |Ax  -  y| |  under  the 
constraint  ||Lx||  =  C,  where  L  is  a  suitably  chosen  linear  operator. 

Equivalently , one  finds 

minimum  {  | |Ax  -  y| |^  +  ||Cxj|^} 

x 

A 

where  y  plays  the  role  of  a  Lagrange  multiplier.  If  L  is  the  identity 


(39) 
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operator,  this  approach  will  result  in  that  solution  x  which  carries  out 

the  minimization  in  (39)  having  a  specified  value  of  | |x| |  .  If  L  is  the 

derivative  operator,  this  approach  will  result  in  that  solution  x  which 

carries  out  the  minimization  in  (39)  having  a  specified  degree  of  smooth- 

2 

ness  (i.e.  a  specified  value  of  ||dx/dt||  ). 

The  parameter  u  is  determined  by  the  constraint  (e.g.  the  specified 
2 

value  of  !|x||  or  the  specified  degree  of  smoothness  or  a  combination  of 
both)  [17]. 

Many  choices  of  the  operator  L  are  possible  and  one  may  simultane¬ 
ously  apply  several  constraints  with  several  corresponding  Lagrange 
multipliers  [17], 

Solution  of  the  problem  posed  in  (39)  is  equivalent  to  the  solution 
of  the  matrix  equation 

(A*A  +  V2  L*L)x  =  A*y  (40) 

and  is  given  by 

x  =  (A*A  +  u2  L*L)_1  A*y.  (41) 

This  approach  is  known  as  the  Tykhonov  regularizing  scheme  [3,18].  Note, 
if  L  is  the  null  operator  (i.e.  if  no  constraint  is  imposed),  the  solu¬ 
tion  in  (41)  reduces  to  the  classical  least  squares  solution  of  (37). 

The  Tykhonov  regularizing  scheme  has  been  used  to  regularize  the 
ill-posed  pattern  synthesis  problem  by  Deschamps  and  Cabayan  [19]  and 
Mautz  and  Harrington  [20]  and  the  image  processing  problem  [23,  24]. 

The  last  approach  to  be  discussed  for  regularizing  an  ill-posed 
problem  is  technique  (e).  In  this  approach  a  stochastic  extension  of 
an  otherwise  deterministic  problem  is  obtained  by  introducing  probabilistic 
concepts.  Once  again,  consider  the  problem  of  Ax  *  y,  where  A  and  y  are 
deterministic  quantities.  As  mentioned  previously,  perfect  measurement, 
or  observation  of  y  is  impossible.  In  some  cases  use  of  the  'noisy" 
measurements  for  y  results  in  an  ill-posed  problem.  With  technique  (e), 
the  problem  is  made  well-posed  by  recognizing  that  uncertainty  in  the 
observation  of  y  causes  uncertainty  in  the  resulting  solution  for  x. 
Consequently,  the  solution  is  viewed  as  a  random  process.  An  error 


n. 
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criterion  is  specified  and  a  stochastically  optimum  solution  is  obtained. 
The  solution  is  stochastically  optimum  in  the  sense  that  repetition  in 
measurements  of  y  produces  solutions  for  x  which,  on  the  average,  are 
optimum  according  to  the  specified  error  criterion. 

Let  the  "noisy"  measurements  of  y  be  denoted  by  d  such  that 

d  =  y  +  n  (42) 

where  n  represents  additive  noise.  In  practice,  n  may  be  due  to  measure¬ 
ment  error,  numerical  roundoff  error,  or  system  noise.  Substitution  of 
(42)  into  (38)  results  in  the  equivalent  problem 

Ax  +  n  =  d.  (43) 

Assume  that  the  means  of  x,  n  and  d  are  known  along  with  the  auto¬ 
correlation  functions  R^,  Rnn,  Rdd  and  the  cross  correlation  functions 

R  ,  and  R  . 
xd  xn 

Without  loss  of  generality,  x,  n  and  d  each  can  be  considered  to 
have  zero  mean  since  an  equation  identical  in  form  to  (43)  is  obtained 
where  the  known  means  are  subtracted  from  the  equation.  If  the  solution 
for  x  is  denoted  by  x,  the  error  is  given  by  c  =  (x  -  x).  For  purposes 
of  illustration,  consider  the  squared-error  criterion  given  by 


lx  -  *! 


(44) 


The  optimum  solution  for  x  is  defined  to  be  that  linear  solution  which 

2 

minimizes  the  mean-squared  error  E[|e|  ]  where  E  denotes  the  expectation 
operator.  Let  the  solution  for  x  have  the  form 

x  -  Ld  (45) 

where  L  is  a  linear  operator.  The  mean-squared  error  is  then  given  bv 

E((x  -  Ld)(x  -  Ld)*]  .  (46) 


E[|e|2] 


In  terms  of  the  known  correlation  functions  the  mean-squared  error 
becomes 

E[  e|  J  -  R  -  LR .  -  R  ,L*  +  LR..L* 

1  1  1  xx  dx  xd  dd 


"  Rxx  "  LRxd  "  Bxd 


L*  +  LRddL* 


-  [R  -R  ,R"Jr  .]  +  [L-R  .RjilR . .[L-R  ,R"1i* 
xx  xd  dd  xd'  xd  dd  dd  xd  dd  • 


(47) 
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Ic  is  concluded  Chat  the  minimum  mean  squared  error  solution  is  given 
by 

x  -  R  r"  d  , 
xd  dd 


Existence  and  stability  of  this  solution  is  guaranteed  when  Rdd  is 
positive  definite. 

Substituting  for  d  from  (43),  the  correlation  functions  in  (49) 

become 

R  ,  “  E[xd*]  =  E[x(x*A*+n*)]  =  R  A*  +  R 
xd  xx  xn 

R , .  ■  E[dd*]  *  E[  (Ax+n)  (x*A*+n*)  ]  =  AR  A*  +  R*  A*  +  AR  +  R__  •  (50) 
do  xx  xn  xn  nn 

Thus  the  solution  may  also  be  expressed  as 

x  =  [R  A*  +  R  ] [AR  A*  +  R*  A*  +  AR  +  R  ]-1d  .  (51) 

xx  xn  xx  xn  xn  nn 

Assuming  the  random  variables  x  and  n  are  uncorrelated,  R  =0.  The 

xn 

solution  then  simplifies  to 

;  ’  R**A*(ARx»A*  -  V'1-1  -  <52) 

Without  noise  R  =0  and  the  solution  reduces  to  the  classical  solution 

-  -1,  nn 

x  *  A  d. 

An  obvious  difficulty  with  (52)  is  that  it  involves  Rxx,  the  auto¬ 
correlation  of  the  deterministic  unknown  in  the  original  equation  (38). 
Therefore,  it  is  necessary  to  assume  a  correlation  function  for  x. 

A 

Franklin  [21]  has  shown  in  a  particular  application  that  x  is  remarkably 

Insensitive  to  the  choice  for  R  . 

xx 

For  the  special  case  in  which 

R_  -  ah  and  *  -  ah  (53) 

xx  x  nn  n 

where  I  is  the  identity  operator  and  the  number  of  unknowns  equals  the 

number  of  equations  available,  then  the  solution  in  (52)  becomes  [22] 

2 

-  °n  -1 

x  -  [A* A  +  — y  I]  A*d  .  (54) 

o 

X 

With  reference  to  (41),  (54)  is  equivalent  to  the  Tykhonov  [4]  -  Philips  [5] 
rwomey  [6-7]  regularization  scheme  when  L  is  the  identity  operator.  It  can 
also  be  shown  that  equivalence  between  (52)  and  (41)  also  holds  when 

Rxx«  (£*t)_1  [26]. 
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4.  Conclusion: 

This  paper  discusses  some  mathematical  concepts  associated  with 
solution  of  the  operator  equation  Ax  =  v.  In  addition  to  illustrating,  by 
examples,  concepts  of  ill-posed  and  well-posed  problems,  various  techniques 
are  presented  for  regularizing  ill-posed  problems. 
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